International Journal of Education Policy & Leadership , December 31, 2014. Volume 9, Number 7. 


Lilli 


5 $ 


School Mental Health: The Impact of State 
and Local Capacity-Building Training 


Sharon Stephan 

University of Maryland School of Medicine 
Carl Paternite 
Miami University 
Lindsey Grimm 

University of Maryland School of Medicine 
& Laura Hurwitz 
Children’s Hospital of Chicago 


Abstract: Despite a growing number of collaborative partnerships between schools and community-based organizations to expand school 
mental health (SMH) service capacity in the United States, there have been relatively few systematic initiatives focused on key strategies 
for large-scale SMH capacity building with state and local education systems. Based on a framework of ten critical factors for capacity 
building, as well as existing best practices, two case studies were utilized to develop a replicable capacity-building model to advance inter¬ 
agency SMH development. Seventy education and mental health stakeholders from two selected states participated in baseline assessments 
of skill competency and critical factor implementation followed by two-day trainings (one in each state); 29 (41%) of the participants also 
completed a six month follow-up assessment. Targeted competencies increased significantly for participants from both states, with large 
effect sizes (d = 2.05 and 2.56), from pre- to post-training. Participant reports of critical factor implementation increased significantly for 
one of the two states (t[ 15] = -6.40,/) < .001, d = 1.77). Results inform specific training recommendations for stakeholders and collabora¬ 
tive teams, as well as policy implications to support future development of SMH service capacity. 
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School mental health; Workforce 
development 

Youth spend roughly half of their waking hours in 
school settings, where it is essential that educators 
provide engaging and supportive learning environ¬ 
ments to promote student academic success (CDC, 
2009; Dinkes, Kemp, & Baum, 2009). However, 


effectively attending to the multitude of students’ 
interrelated academic, social-emotional, and behav¬ 
ioral needs and meeting mandated academic bench¬ 
marks pose difficult challenges for school profes¬ 
sionals. Specifically, one in five school-age youth 
experience emotional and behavioral problems 
(Rones & Hoagwood, 2000), and roughly three- 
fourths of schools identify social, interpersonal, and 
familial problems to be common concerns among 
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their students (Foster et al., 2005). These realities 
highlight the importance of schools’ being equipped 
with the fundamental infrastructure, policies, and 
supports to systematically address student wellbeing, 
including their academic, social-emotional, and be¬ 
havioral needs. The current case studies focused on 
how two states, West Virginia and Utah, implement¬ 
ed a strategy to systematically strengthen the capaci¬ 
ty of local and state level education and community 
partnerships to improve school mental health (SMH) 
programs and services. 

Importance of SMH Programs 
and Services 

Among both education and public health researchers 
and practitioners, there is increasing awareness of 
the critical role of schools in addressing youth men¬ 
tal health, which, as noted above, is inextricably 
linked to academic success (Doll, Spies, & Champi¬ 
on, 2012; Rones & Hoagwood, 2000). However, 
schools are challenged by an overarching mandate to 
raise academic standards, and it is quite easy for 
SMH programs and services to be construed as “add¬ 
ons” that are not vital to the central academic mis¬ 
sion of the educational system. One explanation for 
this problem is that educational administrators and 
teachers often do not feel adequately resourced in 
terms of funding, personnel, or programs to provide 
the mental health supports needed by students in 
their schools (Foster et al., 2005; Weist, Patemite, 
Wheatley-Rowe, & Gall, 2009). Moreover, there is a 
continuing need for development and dissemination 
of effective youth-focused mental health practices, 
and there continues to be a tremendous gap between 
the development and dissemination of evidence- 
based mental health practices and the training, su¬ 
pervision, and infrastructure necessary to implement 
these effective practices in schools (Ringeisen, Hen¬ 
derson, & Hoagwood, 2003; Weist et al., 2009). 

There are a growing number of collaborative 
partnerships between schools and community-based 
health and mental health organizations that are ex¬ 
panding service capacity and making optimal use of 
schools as entry points for an integrated system of 
mental health supports for youth and families. These 
collaborative efforts benefit schools and their part¬ 
nering community agencies through the pooling of 
resources and expertise, movement toward robust 
systems of care, and the development and implemen¬ 
tation of integrated strategies to ensure comprehen¬ 
sive learning supports and to reduce academic and 
non-academic barriers to learning (Weist, Evans, & 


Lever, 2003). Evidence of these collaborations in¬ 
cludes the fact that more than half of schools in the 
United States report having partnered, to some de¬ 
gree, with community-based organizations to pro¬ 
vide mental health services (Foster et al., 2005). De¬ 
spite the growing emphasis on such collaborative 
partnerships, numerous challenges to successful 
partnerships remain (Hooper & Britnell, 2012), and 
there are relatively few studies focused on key strat¬ 
egies for large-scale SMH capacity building with 
state and local education systems. 

Principles of SMH Capacity Building 

Stephan, Hurwitz, Paternite, and Weist (2010) ad¬ 
dressed this gap—that is, the lack of examination of 
key strategies—through research undertaken by the 
School Mental Health-Capacity Build Partnership 
(SMH-CBP). This work was part of a larger project 
and co-operative agreement between the National 
Assembly on School-Based Health Care (NASBHC) 
and the Center for Disease Control Division of Ado¬ 
lescent Health (CDC-DASH; see Figure 1 for SMH- 
CBP timeline). As defined by the CDC, capacity 
building means enabling an organization to operate 
in a more comprehensive, responsive, and effective 
manner via the provision of training, technical assis¬ 
tance, information sharing, materials development, 
technology transfer, or funding (CDC, 2000). SMH- 
CBP activities targeted capacity building of local 
and state education authorities via these mecha¬ 
nisms, with a strategic focus on sustainability, as 
reflected in Flaspohler and colleagues’ (2008) defini¬ 
tion of capacity building as the “dissemination of 
innovations and [the] sustainability of those innova¬ 
tions once they are implemented” (p. 183). 

The purpose of the initial SMH-CBP work was to 
inform the development of a model for SMH capaci¬ 
ty building by conducting site visits in four states. 
The site visits concentrated on how SMH initiatives 
are developed and implemented at state and local 
levels. Maryland, Missouri, Ohio, and Oregon were 
selected based on their track records of success in 
relation to SMH policies and effective SMH practic¬ 
es. For these four states, several key aspects of their 
success were quite evident, including strong state 
level leadership, success in securing federal funding, 
demonstrated cross-agency collaborations, and 
broad, diverse stakeholder investments in advancing 
statewide SMH initiatives. 

Three focus groups were conducted in each of the 
four states and included 119 participants, represent¬ 
ing education leaders, mental health/health system 
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Figure 1. School Mental Health Capacity-Building Partnership Timeline 



leaders, youth development leaders, and family ad¬ 
vocates. Based on structured, iterative content analy¬ 
sis of the focus group data, 10 critical factors for 
capacity building were identified: 

1. Establish a unified, cohesive, and compelling 
vision and a shared agenda with stakeholders— 
one that can inspire local action. 

2. Establish a centralized organizational infrastruc¬ 
ture and accountability mechanisms to ensure 
implementation of the vision and action agenda. 

3. Create feasible and sustainable funding models 
for comprehensive SMH initiatives, including 
early intervention and prevention. 

4. Promote an understanding among state and local 
education leaders of the critical links and associ¬ 
ations between student academic success and 
mental health. 

5. Meaningfully engage diverse family members 
and youth in SMH policy and program devel¬ 
opment. 

6. Recognize the needs of culturally diverse popu¬ 
lations and take steps to reduce disparities in ac¬ 
cess to effective SMH programs and services. 

7. Implement pre-professional and in-service train¬ 
ing for educators and SMH professionals related 
to youth development, youth mental health, and 
best practices in SMH. 


8. Provide support for practitioners in using evi¬ 
dence-based SMH strategies. 

9. Co-ordinate equitable distribution of resources 
and services across schools related to ensuring 
student academic success, mental health, and 
well-being. 

10. Focus on continuous quality improvement by 
collecting and using outcome data to inform de¬ 
cision making at the local school, school district, 
and state levels. 

Purpose of the Current Case Studies 

With the ten critical factors of Stephan et al. (2010) 
as a framework, the purpose of the current case stud¬ 
ies was to fulfill the primary goal of the 
NASBHC/CDC-DASH co-operative agreement: that 
is, to develop a capacity-building model that could 
be replicated by states seeking to advance or build 
cross-agency, systemic SMH efforts. A secondary 
goal was to examine the impact of a two-day train¬ 
ing strategy on participants’ reports of their ability to 
build capacity in SMH. It was hypothesized that par¬ 
ticipants’ self-rated abilities related to SMH capacity 
building would significantly increase from pre- to 
post-training, across training modules. Moreover, 
the trainers were interested in the relation between 
participants’ perceptions of the training as related to 
the implementation of the 10 critical factors six 
months after training. It was hypothesized that par- 
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ticipants’ baseline critical factor implementation 
scores, self-rated abilities across training modules, 
perceived impact of training, and planned use of 
training material would significantly predict their 
mean critical factor implementation scores at the six 
month follow-up. Given that data collected in the 
study is exploratory in nature, we argue that the re¬ 
sults should be viewed in that context, with an un¬ 
derstanding that future work is critical to further de¬ 
velop and test the framework and training. 

Method 

Development of Capacity-Building Train¬ 
ing Modules 

Using the ten critical factors as a guide for determin¬ 
ing training content, the SMH-CBP developed a ca¬ 
pacity-building model that built on existing best 
practices, addressed system-level needs, and was 
relevant to diverse stakeholders. Since developing a 
training component on each of the ten factors was 
not feasible, the SMH-CBP team posed three fun¬ 
damental questions to help home in on specific areas 
of need: (1) Has relevant content on best practices 
already been developed? (2) Is there a critical gap in 
content that needs to be filled? (3) What content is 
absolutely necessary for any state to begin their 
SMH capacity building work? 

Based on these criteria, four modules for capaci¬ 
ty-building training were developed and utilized: (1) 
Overview and Fundamentals of SMH (e.g., defining 
SMH, learning benefits of SMH, understanding the 
connection between SMH and academic success); 
(2) Marketing SMH to School Administrators (e.g., 
identifying steps in social marketing, developing key 
messages about the importance of SMH); (3) SMH 
Quality Improvement (e.g., learning steps in quality 
assessment process, discovering how to develop as¬ 
sessment team); and (4) Statewide SMH Planning 
(e.g., identifying 10 critical factors for advancing 
SMH, prioritizing and identifying action steps for 
state-wide planning). These four key domains of 
training were chosen because they cover necessary 
content for any collaborative teams of educators and 
community partners embarking on SMH efforts, in¬ 
cluding basic information about the SMH field 
(Module 1), social marketing content often not ad¬ 
dressed in existing SMH training and technical assis¬ 
tance efforts (Module 2), training content that em¬ 
phasizes and encourages the use of best practices 
and data-based decision making (Module 3), and 


information and strategies essential to building a 
sustainable capacity-building process (Module 4). 

Separate workgroups were formed to develop 
each of the four training modules. SMH-CBP steer¬ 
ing committee members and other individuals with 
expertise outside of the steering committee (e.g., 
social marketing) participated in the workgroups. 
Each workgroup included at least one family mem¬ 
ber and one representative each from the fields of 
mental health and education. Workgroups met virtu¬ 
ally and in person for several months to develop and 
refine training content. Training modules were pi¬ 
loted in various technical assistance activities and 
were also presented for feedback at national confer¬ 
ences. Evaluation results from these activities were 
used in refining each module. 

Request for Applications (RFA) Process 

Building on lessons learned from the previously de¬ 
scribed site visits in Maryland, Missouri, Ohio, and 
Oregon, and from consideration of the ten critical 
factors, a capacity-building model was developed 
that involved provision of one year of technical as¬ 
sistance with a State Education Agency (SEA) and 
its collaborating local agencies (the two-day SMH 
capacity-building training was embedded early in the 
year of technical assistance). In order to pilot the 
model, the SMH-CBP developed a Request for Ap¬ 
plications (RFA) to solicit applications from state 
departments of education and their partner organiza¬ 
tions 1 . The RFA was sent to state departments of 
education, and nine states submitted applications; 
however, the four “early adopter states” included in 
earlier SMH-CBP site visits were ineligible to apply. 
A review committee comprised of SMH-CBP steer¬ 
ing committee members reviewed and scored the 
applications, using the following scoring criteria: 
existing statewide capacity for SMH (20 points); 
commitment from SEA to advance SMH (15 points); 
readiness to co-ordinate a training initiative (15 
points); intent to implement SMH capacity-building 
activities in the future (20 points); demonstrated 
commitment from partner agencies (15 points); and 
overall merit, quality, and feasibility of the applica¬ 
tion (15 points). All reviewers submitted scores for 
all states, and after the scores were openly discussed 
and reviewed by the team, the state with the highest 
average total score was selected for participation. 
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Implementation and Testing of the SMH 
Capacity-Building Model and Modules 

West Virginia training. Based on being scored 
highest in the RFA review, West Virginia was se¬ 
lected to participate in the pilot initiative. The appli¬ 
cation was submitted by the West Virginia Depart¬ 
ment of Education Office of Healthy Schools in col¬ 
laboration with several partner agencies and organi¬ 
zations, including the Bureau for Behavioral Health 
and Health Facilities; Bureau for Public Health; 
School Health Technical Assistance Center at Mar¬ 
shall University; West Virginia School Based Health 
Assembly; and the Family Advocacy, Support, and 
Training program. Following selection, West Virgin¬ 
ia leaders participated in a series of planning calls 
with NASBHC staff to organize logistics, develop a 
list of invitees, shape the training curriculum and 
identify co-facilitators for the statewide planning 
process. The training took place on May 13-14, 
2009, in Charleston, West Virginia (WV). Six na¬ 
tional experts facilitated and delivered the training to 
a total of 40 SMH stakeholders from a variety of 
agencies and disciplines, including family advocacy, 
education, health, and mental health. 

Based on the results of the training in WV, several 
modifications were made to the training curriculum 
and materials. Experiences and lessons learned from 
the two-day training in WV were built into a second 
RFA announcement presenting a model that incorpo¬ 
rated adaptability to local circumstances and needs 
in selecting training modules. Specifically, the RFA 
specified: (a) that the training could take place in 
one day (with an optional second day); (b) that the 
SEA, in consultation with the SMH-CBP, would 
have the option of selecting specific modules; and 
(c) that the SEA would be required to cover travel 
costs and training stipends for the SMH-CBP facili¬ 
tators. 

Utah training. Two years after submitting their ini¬ 
tial application, Utah responded to the second RFA, 
requesting to receive training and technical assis¬ 
tance from the SMH-CBP using their own funds. 
They requested a two-day training curriculum, in 
fact, with all four training modules. Their RFA ap¬ 
plication was submitted by the Utah State Depart¬ 
ment of Education in partnership with Utah School 
Based Behavioral Health Alliance (USBBHA), Utah 
Division of Substance Abuse and Mental Health 
(DSAMH), Utah Parent Center, National Alliance 
for the Mentally Ill, Allies with Families, and New 
Frontiers for Families, as well as other with local 


school and family partnerships committed to chil¬ 
dren’s mental health. The training took place on 
March 10 and 11, 2011, in Salt Lake City, Utah 
(UT). Two national experts facilitated the workshop, 
and a total of 35 SMH stakeholders attended the 
two-day training. 

Participants 

SMH stakeholders participating in the West Virginia 
and Utah trainings completed an evaluation at the 
end of their two-day trainings and were invited to 
complete evaluations six months after the training 
(“follow-up”). A total of 40 participants in West Vir¬ 
ginia and 31 participants in Utah completed the ini¬ 
tial evaluations during the training (pre and post). 
Using a repeated measures design, 13 West Virginia 
participants from the original sample completed the 
six-month follow-up evaluation, and 16 Utah partic¬ 
ipants completed the six-month follow-up evalua¬ 
tion. All evaluations were anonymous, and follow¬ 
up data were matched to participants’ initials and 
date of birth. 

Measures 

Evaluation of training at baseline. To assess the 
effectiveness of the capacity-building training, at¬ 
tendees were asked to complete short paper-and- 
pencil surveys rating their knowledge of and satis¬ 
faction with the two-day training event across multi¬ 
ple domains. 

Training module skills. Nineteen items measured 
participants’ self-reported abilities pre- and post¬ 
training in each of the four training modules (i.e., 
Overview and Fundamentals of SMH, SMH Quality 
Assessment and Improvement, Marketing SMH to 
School Administrators, and Statewide Planning). 
Responses were rated on a six-point Likert scale 
ranging from 1 (not able) to 6 (totally able). For 
more information on the specific content presented 
in each module, see NASBHC’s website for detailed 
information and trainer materials (www.sbh4all.org). 

Impact of training. Seven items measured partici¬ 
pants’ perceptions of the training event outcomes 
related to specific goals (e.g., “Better understanding 
of how to build capacity in SMH at all levels [build¬ 
ing, local, state]”; “Strengthened my current relation¬ 
ships with SMH partners”). Response options ranged 
from 1 (goal not at all met) to 6 (completely met). 
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Integration of training materials. Four items as¬ 
sessed participants’ perceptions of potential opportu¬ 
nities to integrate training material in the future. 
Questions included, for example, “Do you see oppor¬ 
tunities to integrate material into... (1) developing 
policies, (2) planning efforts, (3) professional devel¬ 
opment, and (4) advocacy/legislative efforts?” For 
each item, participants marked “yes” or “no.” A count 
variable was created, with higher numbers indicating 
they were likely to integrate materials into more pro¬ 
fessional areas. 

Ten Critical Factors. To measure current implemen¬ 
tation in each state (WV and UT) of the 10 critical 
factors identified by Stephan et al. (2010), partici¬ 
pants were asked to rate the degree to which each 
factor was currently being implemented in their 
state. Response options ranged from 1 (not in place) 
to 6 (folly in place). See Stephan et al. (2010) for 
detailed description of the 10 critical factors. 

Evaluation of training at follow-up. Participants 
were asked to complete a six-month follow-up 
online evaluation assessing their perceived use and 
value of the information and materials presented in 
the two-day capacity-building training. Project direc¬ 
tors from each state emailed training participants 
asking them to complete the brief, anonymous sur¬ 
vey. The only items used from the six-month follow¬ 
up survey were participants’ ratings on the 10 critical 


factors. Participants were asked to rate the degree of 
implementation from 1 (not in place) to 6 (fully in 
place) of the 10 critical factors in their state since the 
two-day baseline training. 

Data Analysis 

Separate analyses were conducted for West Virginia 
and Utah data. To examine participants’ self- 
reported ability across all four training modules both 
prior to the training and after the training, paired- 
samples t-tests were conducted using baseline data 
(West Virginia n = 40, Utah n = 30). Likewise, 
paired-samples t-tests were conducted (West Virgin¬ 
ia n = 13, Utah n = 16) to examine if there were sig¬ 
nificant differences in state-wide implementation 
scores of the 10 critical factors from baseline to fol¬ 
low-up. Lastly, a linear stepwise regression was 
conducted to assess if participants’ perceptions of the 
baseline two-day training predicted the mean num¬ 
ber of critical factors in place six months later. Inde¬ 
pendent variables included in the regression equation 
included mean critical factor score at baseline, mean 
self-rated abilities across training modules directly 
post-training, mean impact of training, and mean 
integration of material. 


Table 1. West Virginia Participant Mean Competency Pre- and Post-Baseline Training (n = 40) 



Pre-training 

Post-training 




competency 

competency 




M (SD) 

M (SD) 

t( 39) 

d 

Module 1. 

Overview and Fundamentals of SMH 

Module 2. 

2.89 (1.36) 

4.36 (0.88) 

-10.67** 

1.28 

Marketing SMH to School 
Administrators 

2.85 (1.19) 

4.49 (0.75) 

-9.02** 

1.65 

Module 3. 

Quality Assessment and Improvement 
in SMH 

2.33 (1.16) 

4.42 (0.73) 

-13.37** 

2.16 

Module 4. 

Statewide SMH Planning 

2.48 (1.04) 

4.61 (0.70) 

-13.97** 

2.40 

Modules 1-4: Total Mean Score 

2.74 (1.01) 

4.50 (0.69) 

-15.58** 

2.03 


Note. Paired-samples t-tests were conducted on baseline sample (« = 40) and are presented in column 3. Cohen’s d effect 
sizes are presented in column 4. ** p < .01 
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Results 

West Virginia 

Significant differences were found for participants’ 
mean self-rated abilities across Modules 1-4 be¬ 
tween pre-training (M = 2.74, SD = 1.00) and post¬ 
training (M = 4.50, SD = 0.69): t( 39) = -15.58, p < 
.001. The effect size was also quite large (d = 2.05), 
indicating that participants’ knowledge base for 
SMH capacity building was significantly impacted 
by the training. See Table 1 for mean comparisons of 
West Virginia participants’ self-rated abilities pre- 
and post-training (n = 40). 

For all four modules, there was significant posi¬ 
tive change in participants’ self-rated abilities pre- 
and post-training. An examination of the effect sizes 
for each training module showed that the largest 
changes were seen in participants’ ability to monitor 
effectiveness of SMH programs (Module 3, d = 
2.16) and develop strategies for partnering with var¬ 
ious stakeholders to plan for SMH programming 
(Module 4, d = 2.40) (see Table 1 for all effect siz¬ 
es). 

There were no significant differences in partici¬ 
pants’ overall state-wide implementation scores of 
the 10 critical factors between baseline (M = 2.70, 


SD = 0.47) and follow-up (M = 3.13, SD = 0.91): 
i(ll) = -1.87, /? = .089. However, the effect size was 
moderate (d = 0.59), suggesting this null finding 
may be related to the small sample size. The majori¬ 
ty of the 10 critical factors were rated as being 
slightly higher at six-month follow-up, but not sig¬ 
nificantly different from participants’ rating at base¬ 
line. However, i-tests revealed significant changes in 
implementation of the following critical factors from 
baseline to six months: Factor 4 (connection be¬ 
tween SMH and academic outcomes), Factor 6 (rec¬ 
ognizing the needs of diverse populations), and Fac¬ 
tor 8 (monitoring SMH best practices). Interestingly, 
for Factor 1 (development of shared SMH agenda), 
mean scores slightly decreased at the six-month fol¬ 
low-up assessment. Similarly, for Factor 10 (collect¬ 
ing data on SMH outcomes), participants’ mean 
scores were identical at baseline and at six months. 
Table 3 shows mean comparison for participants’ 
ratings of the 10 critical factor scores at baseline and 
at six-month follow-up assessment. 


Table 3. West Virginia (n = 13) and Utah (n = 16) Mean Implementation 
of Critical Factors at Baseline and Six Months 


West Virginia Utah 


Critical Factors 

Baseline 

M (SD) 

6-month 

M (SD) 

t(12) 

Baseline 

M (SD) 

6-month 

M (SD) 

t(15) 

Factor 1 

3.50 

(0.91) 

3.33 (1.44) 

0.39 

2.44 

(1.15) 

4.06 

(1.29) 

-5.67** 

Factor 2 

3.33 

(1.50) 

3.58 (1.44) 

-0.46 

1.88 

(0.74) 

3.88 

(1.13) 

_7 •ycj** 

Factor 3 

2.00 

(0.85) 

2.25 (1.06) 

-0.638 

1.75 

(0.86) 

3.38 

(1.31) 

-4.78** 

Factor 4 

3.08 

(0.79) 

4.17 (1.03) 

-4.17** 

2.69 

(1.14) 

5.06 

(1.12) 

-6.33** 

Factor 5 

2.33 

(0.78) 

2.50 (1.24) 

-0.48 

1.56 

(0.51) 

3.06 

(1.34) 

-3.98* 

Factor 6 

2.20 

(0.45) 

3.60 (1.14) 

-1.43* 

2.81 

(0.91) 

3.94 

(1.44) 

-3.74* 

Factor 7 

1.82 

(0.75) 

2.46 (1.21) 

-1.47 

1.94 

(0.85) 

3.50 

(1.59) 

-3.65* 

Factor 8 

2.82 

(0.94) 

3.83 (1.27) 

-2.57* 

3.13 

(1.36) 

4.31 

(1.45) 

-3.14* 

Factor 9 

2.42 

(0.79) 

3.17 (1.03) 

-2.14 

2.00 

(0.89) 

3.25 

(1.48) 

-3.73* 

Factor 10 

2.82 

(1.17) 

2.82 (1.08) 

0.00 

2.40 

(1.35) 

3.13 

(1.46) 

-1.59 

Total Mean 

Score 

2.70 

(0.47) 

3.13 (0.91) 

-1.88 

2.26 

(0.54) 

3.75 

(1.06) 

-6.39** 
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Table 2. Utah Participant Mean Competency Pre- and Post-Baseline Training (n = 31) 



Pre-training 

Post-training 




competency 

competency 




M (SD) 

M (SD) 

t(30) 

d 

Module 1. 

Overview and Fundamentals of SMH 

Module 2. 

3.53 (1.29) 

4.88 (0.75) 

-8.22** 

1.28 

Marketing SMH to School 
Administrators 

2.13 (0.96) 

4.34 (0.80) 

-15.14** 

2.50 

Module 3. 

Quality Assessment and 

Improvement in SMH 

2.97 (1.19) 

4.83 (0.73) 

-9.46** 

1.88 

Module 4. 

Statewide SMH Planning 

1.97 (1.04) 

4.78 (0.74) 

-15.64** 

3.11 

Module 1-4: Total Mean Score 

2.66 (0.92) 

4.74 (0.67) 

-15.53** 

2.58 


Note. Paired-samples t-tests were conducted on baseline sample (n = 31) and are presented in column 3. 
Cohen’s d effect sizes are presented in column 4. ** p < .01 


Utah 

Similar to West Virginia, there were significant dif¬ 
ferences in Utah participants’ mean self-rated abili¬ 
ties across Modules 1-4 between pre-training (M = 
2.67, SD = 0.92) and post-training (M = 4.74, SD = 
0.68): t( 29)= -15.53, p < .001. The effect size was 
also quite large (d = 2.56), indicating that partici¬ 
pants’ knowledge base across all four capacity¬ 
building modules was impacted by the training. Ta¬ 
ble 2 depicts Utah’s baseline participants’ (n = 31) 
comparison of training module means at pre- and 
post-training assessment. 

Across all four training modules, significant dif¬ 
ferences emerged in participants’ skill ratings pre- 
and post-training. Effect sizes showed that most no¬ 
table growth was in quality assessment and im¬ 
provement (Module 2, d = 2.50) and statewide SMH 
planning efforts (Module 4, d = 3.11). 

Significant differences were also found in Utah 
participants’ overall statewide implementation scores 
of the 10 critical factors at baseline (M= 2.26, SD = 
0.54) versus follow-up (M = 3.75, SD = 1.06), such 
that participants tended to rate more critical factors 


being in place six months after the training: t(\ 5) = - 
6.40, p < .001. The effect size of 1.77 further indi¬ 
cates that the training had a large impact on partici¬ 
pants’ critical factor scores over time. In contrast 
with West Virginia data, participants reported signif¬ 
icant improvements in critical factors in their state 
for all but one factor, Factor 10 (collecting data on 
SMH outcomes), which did not significantly change. 
Table 3 shows mean comparison for participants’ 
ratings of the 10 critical factors scores at baseline 
and at six-month follow-up assessment. 

Results of the linear stepwise regression revealed 
that participants’ ratings of Utah’s 10 critical factors 
were negatively predicted by their integration of 
training materials: F(l,12) = 7.83, p < .05. Specifi¬ 
cally, higher participants’ reports for planning on 
integrating training materials into their professional 
work predicted lower ratings of the 10 critical fac¬ 
tors. Interestingly, participants’ mean score for the 
10 critical factors at baseline (p = 0.25), post¬ 
training mean scores on the four training modules (p 
= 0.06), and mean impact of training (p = 0.16) were 
not significantly related to how many critical factors 
participants perceived to be in place in their state. 
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Discussion 

At the local level, there is increased emphasis on 
collaborative partnerships between schools and men¬ 
tal health provider organizations to make optimal 
use of schools as entry points in an integrated system 
of mental health care for school-aged youth (Kazak 
et al., 2010; Shapiro, DuPaul, Barnabas, Benson, & 
Slay, 2010). These collaborative efforts benefit 
schools and the collaborating community agencies 
through the pooling of resources and expertise, 
movement toward true systems of care, and the de¬ 
velopment and implementation of integrated strate¬ 
gies to reduce academic and non-academic barriers 
to learning for students (Bringewatt & Gershoff, 
2010; Cappella, Frazier, Atkins, Schoenwald, & 
Glisson, 2008; Weist, Evans, & Lever, 2003). Yet, 
such collaboration is not consistent at the school 
building, school district and/or state levels. The cur¬ 
rent case studies provide a framework for how state 
and local education agencies (SEAs and LEAs) can 
build capacity for a full continuum of mental health 
promotion, prevention, and intervention programs 
and services offered in schools, facilitated by 
statewide trainings. Key implications for policy 
makers, policy implemented, SEAs, and LEAs, as 
well as lessons learned from the initiative are de¬ 
scribed below. 

Implications for Statewide Capacity- 
Building Trainings 

In addition to the four states that participated in the 
pilot focus group study preceding the current case 
studies, ten states submitted applications for the two- 
day capacity-building training. Thus, it appears that 
there is a growing interest at the state level for this 
type of professional training. The current case stud¬ 
ies represent one of the first statewide initiatives to 
be systematically implemented and evaluated. In 
general, participants reported that the training posi¬ 
tively impacted their ability to build capacity in 
SMH, helped build and strengthen their relationships 
with SMH partners, and increased their commitment 
to SMH services across the state. Notably, the two 
most impactful aspects of the training were: (1) iden¬ 
tifying assessment tools that measure organizational 
change and quality improvement in SMH programs 
(Module 3), and (2) understanding how to engage in 
system-level planning (Module 4). These finding fit 
with and highlight the relevance of prior research 
suggesting that ongoing data-based decision-making 
is needed when making large-scale organizational 
changes (Fixsen, Blase, Horner, & Sugai, 2009; 


Hoyle, Samek, & Valois, 2008; Rossi, Lipsey, & 
Freeman, 2004). Based on the current findings, 
stakeholders are encouraged to integrate quality as¬ 
sessment evaluation tools throughout the planning, 
organization, and benchmarking processes. A variety 
of tools are available to assist in this evaluative pro¬ 
cess, and a combination of these may prove the most 
useful for understanding progress at different stages 
of the change process. For instance, collaborative 
teams can use the Organization Readiness for 
Change Assessment (Lehman, Greener, & Simpson, 
2002) during the planning stage, the Strengthening 
Partnerships: Community School Assessment Check¬ 
list (Blank & Langford, 2000) to assess project 
stakeholders’ investment, and the Mental Health 
Planning & Evaluation Template (see 
www.nasbhc.org) to systematically evaluate the 
quality of SMH services across levels. 

As noted previously, significant positive changes 
were seen in participants’ knowledge of the 10 criti¬ 
cal factors for advancing SMH. However, it remains 
unclear whether the knowledge gained during the 
training significantly impacted the state’s implemen¬ 
tation of these critical factors. West Virginia partici¬ 
pants’ implementation ratings of the 10 critical fac¬ 
tors were not significantly different between baseline 
and the six-month follow-up, though numerical 
changes were noted for several individual critical 
factors. Conversely, Utah participants’ scores signif¬ 
icantly increased across all domains except for Fac¬ 
tor 10. In addition, West Virginia participants tended 
to perceive their state to have more of the 10 critical 
factors in place at baseline as compared to Utah, and 
there was a relatively small number of West Virginia 
follow-up respondents, both of which could account 
for the null finding. Future evaluation of similar 
statewide training efforts may benefit from mixed- 
method designs incorporating focus groups and/or 
key informant interviews to further inform the find¬ 
ings. For instance, it would be helpful to understand 
from respondents what strategies were successful in 
advancing the critical factors and, conversely, what 
obstacles impeded progress. In addition, the variabil¬ 
ity between states/districts in baseline responding on 
the 10 critical factors speaks to the need to individu¬ 
alize the training process. Although the team set out 
to create a “standardized” set of training modules, it 
was always with the understanding that certain mod¬ 
ules and even components of modules may be more 
relevant for some states/districts than others, and 
therefore should be applied accordingly. This, in 
fact, is how the framework and materials have been 
used to date. 
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Implications for Policy Makers 

The current case studies have implications for policy 
makers at the local, state, and federal levels. Re¬ 
search has shown that school-based health and men¬ 
tal health services enhance access to mental health 
care for students (Keeton, Soleimanpour, & Brindis, 
2012), help reduce stigma (Nabors & Reynolds, 
2000), and are associated with several positive youth 
outcomes, including improvements in emotional 
functioning, decreases in functional impairment, and 
the potential to improve academic performance 
(Kutash, Duchnowski, & Green, 2011). However, 
little research has been done on how states can build 
capacity to foster these initiatives. As outlined by 
Stephan et al. (2010), building SMH capacity across 
systems is a multifaceted and often overwhelming 
task for stakeholders. The two-day trainings detailed 
in the current case studies streamlined the capacity¬ 
building process for participants by offering training 
and resources in areas identified as critical by other 
leading states, and then having participants system¬ 
atically identify their existing resources (e.g., leaders 
in their state, infrastructure, existing partnerships) 
and potential challenges (e.g., lack of funding, poor 
buy-in) before embarking on a statewide action 
planning process. The State Implementation and 
Scaling-up of Evidence-based Practices (SISEP) 
Center identified similar areas of need for states 
when developing plans for comprehensive change. 
Specifically, the SISEP research team suggests that 
states consider and understand the risks and benefits 
required from changes, create clear communication 
and avenues for feedback across partners, and initi¬ 
ate on-going training and professional development 
to support teachers and staff (Fixsen et al., 2009). 

The current case studies, and the SMH-CBP 
more broadly, reflect the growing trend for federal 
agencies to favor programs that embed cross- 
collaborative partnerships as a key element. For ex¬ 
ample, the Centers for Disease Control and Preven¬ 
tion (CDC) and Division of Adolescent and School 
Health (DASH) have continuously supported the 
Co-ordinated School Health (CSH) approach, a 
guiding framework for school health care delivery 
and one that highlights the importance of system 
integration. Recent research has supported the CSH 
model, with findings showing gains in academic 
achievement (Rosas, Case, & Tholstrup, 2009), en¬ 
hanced collaboration across disciplines (Berzin, 
O’Brien, & Tohn, 2012), and improved school health 
infrastructure (Stoltz, Coburn, & Knickelbein, 
2009). Strong support for a multi-disciplinary, cross¬ 
agency approach to SMH is also found in influential 


reports from the Office of the Surgeon General and 
President George W. Bush’s New Freedom Commis¬ 
sion on Mental Health, which affirmed the un¬ 
matched ability of public schools to provide an ac¬ 
cess point for recognizing and addressing students’ 
mental health needs (U.S. Department of Health and 
Human Services, 1999; New Freedom Commission 
on Mental Health, 2003). By bringing together a 
multi-disciplinary group of stakeholders from across 
each participating state, the capacity-building pro¬ 
cess ensured a diversity of perspectives and re¬ 
sources, with the ultimate goal of fostering a richer 
and more meaningful statewide agenda with cross¬ 
agency investment. Of note, despite having 
statewide family advocacy organizations at the ca¬ 
pacity-building trainings, leaders from one partici¬ 
pating state did not have the continued family en¬ 
gagement they had hoped for, and have since been 
actively working to foster System of Care principles 
of family-driven care into all of their continued 
SMH planning efforts. 

Limitations 

There are several important limitations to note re¬ 
garding the current findings. For one, data are based 
on participants’ self-reports, so social desirability or 
other biases may have influenced responses. Alt¬ 
hough steps were taken to ensure de-identification of 
data, and to encourage participants to be honest in 
their responses in order to fully inform a quality im¬ 
provement process, it is reasonable to consider that 
responding may have been biased due to the fact that 
the trainers were also collecting the data. Second, 
rating scales included compilation of single-item 
indicators previously used in implementation re¬ 
search, resulting in essentially an index of selected 
items. Thus, additional research is needed to exam¬ 
ine the validity, reliability, and factor structure of 
these indices to better understand the strength of 
their psychometric properties as scales. Third, there 
was a large attrition rate for both states between 
baseline and follow-up assessment, with only 33 
percent of the West Virginia sample and 53 percent 
of the Utah sample responding to the six-month fol¬ 
low-up survey. It is possible that participants who 
felt strongly about the training, either positively or 
negatively, responded to the follow-up survey re¬ 
quest. Thus, the data should be interpreted with cau¬ 
tion. Fourth, the case studies and associated data 
reflect only two states involved in the capacity¬ 
building training, so responses may not generalize 
nationally across all states. In sum, these data are 
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exploratory, based on two state-level case examples 
examining the process and outcomes involved in 
building their SMH capacity. Additional research is 
needed to further examine the impact of capacity¬ 
building trainings on participants’ abilities to devel¬ 
op and sustain SMH programs across their state. Fi¬ 
nally, it is important to note that while training is 
important for developing and enhancing state and 
local capacity to achieve quality school mental 
health, there are certain factors within the framework 
that will be less impacted by such training efforts, at 
least within the time frame of the current study. For 
example, although training in system planning may 
help stakeholders become more aware of their needs, 
without the provision or reallocation or funds, the 
training alone will not create a feasible and sustaina¬ 
ble funding model. 

Conclusion 

The task of capacity building in an effort to create 
systemic change within an organization, especially a 
school setting, can be an arduous task for educators 
and policy makers alike, as it often requires cross¬ 
system collaboration, organizational readiness for 
change, and the mobilization of innovation. The 
nested nature of schools (i.e., students nested within 
classrooms within schools embedded in communi¬ 
ties) requires program stakeholders to be proactive 
and take a wider lens when developing an integrative 
and innovative capacity-building model (Cosner, 
2009; Massell, 2000). Yet, there is a paucity of re¬ 
search identifying the key components of school- 
based capacity building as it specifically relates to 
children’s mental health in educational settings. The 
current case studies contribute to a needed area of 
research by illustrating how comprehensive training 
in SMH capacity building can positively affect 
statewide implementation of needed SMH programs. 
In the interest of supporting feasible local and state 
SMH capacity-building efforts, all materials devel¬ 
oped as part of these case studies, within the context 
of the larger SMH-CBP initiative, have been de¬ 
signed as self-directed modules and are available 
free of charge (see www.sbh4all.org or 
csmh.umaryland.edu). 

Note 

1. Follow-up interviews with focus group partici¬ 
pants subsequent to the site visits to the early 
adopter states revealed that just the process of bring¬ 
ing together stakeholders to share perspectives on 


SMH had a benefit both to the individuals who par¬ 
ticipated and to the states’ efforts as a whole. There¬ 
fore, mirroring the site visit process, an expectation 
of the selected state was to convene a diverse range 
of stakeholders from key state and local agencies 
(i.e., departments of education, mental health/human 
services, physical health) and from child-serving 
organizations, as well as youth and their families, to 
participate in the two-day SMH capacity-building 
training. 

References 

Berzin, S., O’Brien, K„ & Tohn, S. (2012). Work¬ 
ing on what works: A new model for collabo¬ 
ration. School Social Work Journal, 36(2), 15- 
26. 

Blank, M.J., & Langford, B.H. (2000). Strengthen¬ 
ing partnerships: Community school assess¬ 
ment checklist. Washington, DC: Coalition for 
Community Schools. Retrieved from 
www.communityschools.org/CCSDocuments/ 
assessmentnew.pdf 

Bringewatt, E., & Gershoff, E. (2010). Falling 
through the cracks: Gaps and barriers in the 
mental health system for America’s disadvan¬ 
taged children. Child Youth Sendees Review, 
32, 1291-1299. 

Cappella, E., Frazier, S.L., Atkins, M.S., Schoen- 
wald, S.K., & Glisson, C. (2008). Enhancing 
schools’ capacity to support children in pov¬ 
erty: An ecological model of school-based 
mental health services. Administration and 
Policy in Mental Health, 35, 395-409. 

Centers for Disease Control and Prevention (CDC). 
(2000). Building capacity—technology trans¬ 
fer efforts and sustainability for HIV preven¬ 
tion. [Program briefing]. Atlanta, GA: National 
Center for HIV, Viral Hepatitis, STD, and TB 
Prevention. 

Centers for Disease Control and Prevention (CDC). 
(2009). School connectedness: Strategies for 
increasing protective factors among youth. At¬ 
lanta, GA: U.S. Department of Health and 
Human Services. 

Cosner, S. (2009). Building organizational capacity 
through trust. Educational Administration 
Quarterly, 45(2), 248-291. 



SCHOOL MENTAL HEALTH 


12 


Dinkes, R., Kemp, J., & Baum, K. (2009). Indica¬ 
tors of school crime and safety: 2008. Wash¬ 
ington, DC: National Center for Education Sta¬ 
tistics, Institute of Education Sciences, U.S. 
Department of Education; Bureau of Justice 
Statistics, Office of Justice Programs, U.S. 
Department of Justice. NCES 2009-022/NCJ 
226343. 

Doll, B., Spies, R., & Champion, A. (2012). Con¬ 
tributions of ecological school mental health 
services to students’ academic success. Journal 
of Educational and Psychological Consulta¬ 
tion, 22, 44-61. 

Fixsen, D.L., Blase, K.A., Homer, R., & Sugai, G. 
(2009). Scaling up brief 3: Readiness for 
change. Chapel Hill, NC: University of North 
Carolina, SISEP project, FPG Child Develop¬ 
ment Institute. 

Flaspohler et al. (2008). Unpacking Prevention Ca¬ 
pacity: An Intersection of Research-to-practice 
Models and Community-centered Models. 
American Journal of Community Psychology 
41. pp.182-196. 

Foster, S., Rollefson, M., Doksum, T., Noonan, D., 
Robinson, G., & Teich, J. (2005). School men¬ 
tal health services in the United States, 2002- 
2003. Rockville, MD: Center for Mental 
Health Services, Substance Abuse and Mental 
Health Services Administration. DHHS Pub. 
No. (SMA) 05-4068. 

Hooper, L.M., & Britnell, H.B. (2012). Mental 
health research in K-12 schools: Translating a 
systems approach to university-school partner¬ 
ships. Journal of Counseling & Development, 
90, 81-90. 

Hoyle, T.B., Samek, B.B., & Valois, R.F. (2008). 
Building capacity for the continuous im¬ 
provement of health-promoting schools. Jour¬ 
nal of School Health, 78, 1-8. 

Kazak, A.E., Hoagwood, K., Weisz, J.R., Hood, K., 
Kratochwill, T.R., Vargas, L.A., & Banez, 
G.A. (2010). A meta-systems approach to evi¬ 
dence-based practice for children and adoles¬ 
cents. American Psychologist, 65, 85-97. 

Keeton, V., Soleimanpour, S., & Brindis, C. 
(2012). School-based health centers in an era 
of health care reform: Building on history. 
Current Problems in Pediatric and Adolescent 
Health Care, 42, 132-156. 


Kutash, K., Duchnowski, A., & Green, A. (2011). 
School-based mental health programs for stu¬ 
dents who have emotional disturbance: Aca¬ 
demic and social-emotional outcomes. School 
Mental Health, 3, 191-208. 

Lehman, W.E.K., Greener, J.M., & Simpson, D.D. 
(2002). Assessing organizational readiness for 
change. Journal of Substance Abuse Treat¬ 
ment, 22, 197-209. 

Massell, D. (2000). The district role in building ca¬ 
pacity: Four strategies. [Policy Brief]. Phila¬ 
delphia, PA: Consortium for Policy Research 
in Education. ERIC Document Reproduction 
Service No. ED453575. 

Nabors, L. A., & Reynolds, M. W. (2000). Program 
evaluation activities: Outcomes related to 
treatment for adolescents receiving school- 
based mental health services. Children's Ser¬ 
vices: Social Policy, Research, and Practice, 
3. pp. 175-189. 

New Freedom Commission on Mental Health. 
(2003). Achieving the promise: Transforming 
mental health care in America (Final report). 
Rockville, MD. DHHS Pub. No. SMA-03- 
3832. Retrieved October 13, 2014, from 
http://govinfo.library.unt.edu/mentalhealthcom 
mission 

Ringeisen, H., Henderson, K., Hoagwood, K. 
(2003). Context matters: Schools and the “re¬ 
search to practice gap” in children’s mental 
health. School Psychology Review, 32, 153— 
168. 

Rones, M., & Hoagwood, K. (2000). School-based 
mental health services: A research review. 
Clinical Child and Family Psychology, 3, 223- 
241. 

Rosas, S., Case, J., & Tholstrup, L. (2009). A retro¬ 
spective examination of the relationship be¬ 
tween implementation quality of the co¬ 
ordinated school health program model and 
school-level academic indicators over time. 
Journal of School Health, 79(3), 108-115. 

Rossi, P.H., Lipsey, M.W., & Freeman, H.E. 
(Eds.). (2004). Evaluation: A systematic ap¬ 
proach. Thousand Oaks, CA: Sage. 

Shapiro, E.S., DuPaul, G.J., Barnabas, E., Benson, 
J.L., & Slay, P.M. (2010). Facilitating school, 
family, and community partnerships: Enhanc¬ 
ing student mental health. [Special Issue]. 
School Mental Health, 2, 45-51. 



STEPHAN, PATERNITE, GRIMM, & HURWITZ 


13 


Stephan, S., Hurwitz, L., Patemite, C., & Weist, M. 
(2010). Critical factors and strategies for ad¬ 
vancing statewide school mental health policy 
and practice. Advances in School Mental 
Health Promotion, 3, 48-58. 

Stoltz, A., Cobum, S., & Knickelbein, A. (2009). 
Building local infrastructure for co-ordinated 
school health programs: A pilot study. The 
Journal of School Nursing, 25(2), 133-140. 

U.S. Department of Health and Human Services. 
(1999). Mental health: A report of the Surgeon 
General. Rockville, MD: U.S. Department of 
Health and Human Services, Substance Abuse 
and Mental Health Services Administration, 
Center for Mental Health Services, National 
Institutes of Health, National Institute of Men¬ 
tal Health. 

Weist, M.D., Evans, S.W., & Lever, N. (2003). 
Handbook of school mental health: Advancing 
practice and research. New York, NY: Kluwer 
Academic/Plenum Publishers. 

Weist, M.D., Paternite, C.E., Wheatley-Rowe, D., 
& Gall, G. (2009). From thought to action in 
school mental health. International Journal of 
Mental Health Promotion, 11, 32-41. 


IJEPL is a joint publication of PDK International, the Faculty of Education at Simon Fraser University and the College of 
Education and Human Development at George Mason University. By virtue of their appearance in this open access journal, 
articles are free to use, with proper attribution, in educational and other non-commercial settings 90 days after initial publi¬ 
cation. Copyright for articles published in IJEPL is retained by the authors. More information is available on the IJEPL 
Web site: http://www.ijepl.org 





